Establishment of a no-notice drill mode evaluation system for public health emergencies

Objective At present, there are some no-notice drill mode evaluation systems for public health emergencies in Chinese hospitals, which are the subjects of assessment in this study. However, there is a lack of CDC. This study builds a set of no-notice drill mode evaluation systems for public health emergencies that involve the CDC. Methods The indexes for these systems were based on the performance of two no-notice drills for public health emergencies in Guangdong Province. Twenty experts were invited to screen the indicators during two rounds of the Delphi method to determine the weight of first- and second-level indexes through the analytic hierarchy process, and the weight of the third-level index was calculated using the percentage method. Results After two rounds of expert consultation, we obtained four first-level indicators, twenty-six second-level indicators and eighty-six third-level indicators. According to the weight calculated by analytic hierarchy process, the weights of the first-level indicators are emergency preparation (0.2775), verification and consultation regarding an epidemic situation (0.165), field investigation and control (0.3925) and summary report (0.165). Sensitivity analysis shows that the stability of the index is good. Conclusion The no-notice drill mode evaluation system for public health emergencies constructed in this study can be applied to public health departments such as the CDC. Through promotion, it can provide a scientific basis for epidemiological investigation assessment.


Introduction
In recent years, the new coronavirus pneumonia pandemic has swept the world. To test and evaluate the capacity of health emergency teams to respond to public health emergencies, many provinces and cities in China have strengthened emergency drills. The drill for public health emergencies can clarify the responsibilities and tasks of personnel at all levels, train the health emergency team, and take correct actions in the response process. At present, most emergency drills in China are conducted according to a drill script, which is helpful in familiarizing individuals with the process and improving emergency preparedness.
However, due to the particularity and complexity of public health emergencies, the key to managing such a public health crisis is epidemiological investigation and control of disease spread. If we make mistakes in judgment, it may lead to an epidemic or pandemic. Therefore, in emergency drills, we should focus on strengthening epidemiological investigations and the ability to control disease spread. The no-notice drill is exactly the direction these drills should take. The party participating in the exercise should not know the epidemic scenario in advance. Participants in the exercise can only assess the situation after completing the investigation on site. This achieves the purpose of assessment. To construct the no-notice drill mode evaluation system for public health emergencies, the Delphi method and analytic hierarchy process (AHP) were used in this study.

Problem description
In the available literature, research on no-notice drills for public health emergencies at home and abroad is relatively limited: most studies focus on emergency disposal and emergency treatment of mass casualties [1][2][3], mass evacuation [4] and no-notice drill of mass vaccination [5]. In addition, there are studies on no-notice drills for public health emergencies, such as the Ebola no-notice drill held in Taiwan in 2014 [6] and a no-notice drill held in New York City for respiratory infectious diseases such as measles and influenza in 2015 [7]. However, these exercises mainly evaluated the hospital's emergency preparedness for public health emergencies and failed to assess the ability of epidemiological investigators in an epidemic situation4. In addition, researchers have not thoroughly studied the evaluation system for the no-notice drill.
In addition, the existing research related to the no-notice drill for public health emergencies is mainly limited to the implementation of the drill, the evaluation process and result analysis, and there is a lack of the construction process of the evaluation system. The evaluation system of the no-notice drill has not been deeply studied. The construction of these drill evaluation systems mainly relies on some existing drill guidelines, such as the "Hospital Surge Evaluation Tool" used to evaluate the emergency response capacity of mass casualties [3], the Homeland Security Exercise and Evaluation Program (HSEEP) [5,7], the hospital evaluation standards and hospital infection control guidelines issued by China [6], and some of them adopt the simple Delphi method [4]. Compared with the Delphi-AHP, these studies are arbitrary and lack scientificity in the selection of indicators, especially in the determination of weight, which is not conducive to the evaluation of emergency capacity.
To solve the two key problems mentioned above, improve the epidemiological investigation ability of personnel in public health departments, and develop scientific evaluation tools, this study improves the existing no-notice drill mode evaluation system for public health emergencies and develops a set of evaluation tools suitable for flow investigators. The indexes are screened, and the weight is determined by the Delphi method and AHP.
The Delphi method is an effective group consensus consultation method that is widely used in the fields of medicine and public health [8][9][10]. It includes a literature review, stakeholder ideas and expert judgment. The research results are designed and collected by an anonymous expert consultation questionnaire [11][12][13], which has high reliability. Because the Delphi method is mainly aimed at qualitative research [14,15], it is often combined with the AHP in qualitative and quantitative research [16][17][18][19].
The AHP was proposed by Thomas Saaty (1980). To date, this method has undergone many modifications. In recent research, to overcome some defects of this method, the AHP has been combined with fuzzy logic theory [20,21], which models basic information and approximations [20][21][22]. In addition, in view of the excessive number of pairwise comparisons, many experts have also made modifications on the basis of the AHP [23,24] and formulated the BWM, FUCOM [25] and other methods, especially the BWM, which has been applied often in recent years [26,27]. However, in some cases, the AHP is still used in its original form [28,29]. The AHP is widely used in the construction of evaluation systems [30][31][32]. By comparing the opinions of experts, the quantitative relationship between the elements of the same level and the elements of the upper level is determined to assign the relative important weight of the lowest level (schemes and measures for decision-making) relative to the highest level (overall goal).

Delphi method
The Delphi method can be applied to the establishment of various evaluation index systems and the determination process of specific indicators. Through several rounds of feedback, we made full use of and absorbed the experience and knowledge of experts so that the opinions of the experts gradually converged. In this study, we planned to invite approximately twenty domestic experts in the field of public health to screen and revise the indicators through two rounds of the Delphi method to determine the indicators and weights (see Fig 1).
The first round constructed evaluation indicators. We considered the basic steps of infectious disease outbreak investigation [33,34], the guide to health emergency drills prepared by the central disease control, the emergency plan and technical scheme for the emergency disposal of public health emergencies in Guangdong Province, and the framework of the simple scoring table of the two no-notice drills held in China in 2015 and 2016 (formulated under the guidance of the emergency management experts of Guangzhou Center for disease prevention and control) [35]. This round was modified and developed into a preliminary framework of evaluation indicators.
Before issuing the questionnaire, we invited twenty-six domestic public health experts to participate in our study, and twenty agreed to participate. Participants were asked to form an expert group via text messages and e-mail, and the group included university professors from Southern Medical University and the School of Public Health of Sun Yat sen University, experts and managers who have long been engaged in front-line treatment of infectious diseases from Guangdong Provincial Health Commission, Guangdong Emergency Hospital, Guangdong Provincial and Municipal CDC. Among them, sixteen had participated in at least one no-notice drill and were responsible for the participants, evaluation team and expert group in the drill. Therefore, they have a certain understanding of this drill mode.
There are two rounds of Delphi consultation. In the first round, the expertise of participants and their familiarity with the disease scenario are evaluated, giving us the expert authority coefficient (Cr). At the same time, experts score the importance of the first-and second-level indexes [15]. The score is divided into five levels from high to low according to a Likert scale (5 points are very important and 1 point is very unimportant). Considering that it is difficult to carry out and assess the no-notice drill, the third level index not only evaluates the importance score but also scores the feasibility. The scoring standard is also divided into five levels from high to low according to the Likert scale. After collecting data from the first round, the coefficient of variation CV of the third-level indicators were calculated, and the indicators that cannot meet the importance (or feasibility) 16 assignment mean�3.65 and the coefficient of variation < 0.25 were eliminated. At the same time, open suggestions were taken, and suggestions mentioned by at least two experts were selected as new indicators.
In the second round, the revised indicators were distributed. The distribution object was the experts who provided feedback in the first round, and the evaluation content included the importance scores, the feasibility score and the variation coefficient of the third-level indicators. The indexes that cannot meet the mean value of importance and feasibility assignment �3.65 and the coefficient of variation < 0.25 were eliminated. The average value of the importance assignment of the first-and second-level index was transformed into a judgment matrix, and the weight of each first-and second-level index was calculated by the AHP.
This study uses an AHP to determine the weight of evaluation indicators of health emergency drills, which mainly follows these steps:

Establish hierarchical model
This is generally divided into two layers: the top layer is the target layer, and the bottom layer is the standard layer.

Construct judgment matrix
The values of judgment matrix elements reflect people's understanding of the relative importance of various factors. Generally, the judgment matrix is constructed by pairwise comparison between indicators, specifically using the 1~9 scale of scholar GWM van der Staay (see Table 1). According to the Staay scale, the judgment matrix is constructed. The first-level index and the second-level index were compared to calculate the weight Wi of each index.
(1) First, the eigenvector of the judgment matrix is determined, which is also the relative weight of each factor.
Finally, the consistency of the indicators is tested. First, calculate the maximum eigenvalue of the judgment matrix: Recalculate the consistency index: CI ¼ l max À n nÀ 1 . The random consistency index RI can be obtained by looking up the table. Consistency test: If the consistency ratio of the discriminant matrix CR<0. 1, the consistency of the judgment matrix is qualified [16].
After the importance of the third-level indicators is assigned and the corresponding score is calculated by the percentage assignment method, the final weight of each third-level indicator is calculated by the percentage method [15].
The index system validity evaluation evaluates content validity and structure validity. Content validity mainly depends on the correctness of the whole research method and step calculation process. In this study, on the basis of reviewing the literature, we formulated the evaluation framework and index content selection criteria. Then, two rounds of Delphi expert consultation were conducted to select and modify indicators. The evaluation index includes the basic content to be evaluated.
The indicator system reliability evaluation uses Cronbach's alpha coefficient to evaluate the internal reliability of the index system, and α>0.80 was the criterion for determining the reliability of the index. Finally, sensitivity analysis was carried out by changing the weight coefficient of the criterion.

Basic information
In this study, all experts had high academic achievements in their respective fields. Nineteen (95%) were provincial and municipal experts, 19 (95%) had graduate-level educations or more, and 19 (95%) had senior deputy titles or above. They are front-line personnel or emergency management experts who had been engaged in public health work for an average of 23.9 (12-44) years (see Table 2). This round demonstrated that the basic advice from experts was helpful. In this study, twenty questionnaires were distributed in two rounds of the Delphi survey, and nineteen were recovered, for an effective recovery rate of 95%. The questionnaire recovery rate was high. The positive coefficient of the two rounds of experts was 95%. The expert authority coefficient (CR) was 0.805 (> 0.8), indicating that the expert consultation results were accurate and reliable [36].
Concentration of expert opinions. In this study, the concentration and coordination of expert opinions (average score and coefficient of variation CV of third-level indicators) of each index were calculated. After two rounds of expert consultation, the average comprehensive score of third-level indicators increased from 4.29 (3.99 to 4.74) to 4.56 (4.25 to 4.80), and the concentration of expert opinions increased significantly.
Index screening results. In the first round of the survey, the average CV was 0.19 (0.10 to 0.31), and four third-level indicators were greater than 0.25. According to the scores and opinions of experts, one second-level indicator and seven third-level indicators were removed, and one second-level indicator and four third-level indicators were added. In the second round of the survey, the average coefficient of variation was 0.10 (0.06 to 0.21), and no index was greater than 0.25, indicating that the opinions of experts tended to be consistent [37].
Based on two rounds of expert opinions, a no-notice drill evaluation index system composed of four first-level indicators, twenty-six second-level indicators and eighty-six thirdlevel indicators was finally formed. The main indicators were as follows: (1) emergency preparedness: preparation of personnel, materials and plans; information transmission and response speed; (2) epidemic situation verification: verification and preliminary investigation of the incident; and (3) field investigation and control: case epidemiological investigation, external environment sampling, preliminary report and information release; (4) Summary report: whether the content of the investigation report is comprehensive and whether there is discussion, summary and reflection.
Weight of indicators. The AHP and percentage method were used to determine the weights of various indicators of the no-notice drill evaluation system for public health emergencies, as shown in Table 3. The weights of the four first-level indexes were emergency preparation (0.2775), verification and consultation regarding an epidemic situation (0.165), field investigation and control (0.3925) and summary report (0.165). Among them, the field investigation and control subindexes were ranked the highest, and their weight was the heaviest.
The validity of the index system is as follows: (1) Content validity: According to the Delphi expert consultation method, there are eighty-six third-level indicators in the final index system. The average score of each index is 4.65 (4. , the average CV is 0.12 (0-0.21), and the average percentage of full marks is 71.27% (41.18-100.00%) ( Table 1). This shows that the content validity is good. (2) Index system reliability: In the total index system, the alpha reliability coefficient of the eighty-six indexes is 0.989>0.8, and the reliability is high. The alpha reliability coefficients of the internal indexes of the four major links are 0.939, 0.952, 0.988 and 0.902, indicating that the consistency of the indexes of the no-notice drill is good.

Case analysis
To compare the implementation of some public health emergency drills in China, eight public health emergency comprehensive drills A1-A8 (see Table 4) with published papers and public data were selected, and the ranking of A1-A8 was calculated based on the new evaluation system constructed by the Delphi AHP in this study. The method is to score and rank the eight drills according to the weight C1-C4 of four primary indicators (see Table 5). Among them, C3 (field investigation and control) is listed as the most important index, with a weight of 0.3925.

Sensitivity analysis
In recent years, the sensitivity analysis of AHP has mostly been carried out through the change in the criterion weight coefficient, and the criterion selection generally only selects the index  Table 3. Evaluation index and weight of no-notice drills for public health emergencies.

First-level index weight Second-level index Third-level index Weight
Field investigation and control 0.3925 Whether the field command has a reasonable division of labor in the field 0.0427 Send a survey of professionals to each scene 0.0221 The team consists of an investigation group, a sterilization group, an inspection group, a health education group, a logistic support group and a medical group (the teams are grouped according to the situation) Determine the need to carry out guidance for high-risk places (hospitals, etc.) and control suggestions for epidemic situation.

0.0019
Provide health education for patients, close contacts and high-risk groups. 0.0018 Preliminary report on the epidemic situation and judgment to the health administration department 0.0617 The report includes the following: preliminary judgment of events, preliminary control suggestions, problems and problems to be solved.

0.0617
Public information release and media response 0.0137 Deal with the media correctly and communicate moderately in time. 0.0068 Have a designated press spokesperson. 0.0069 Emergency termination and aftermath 0.0137 According to the development of the incident and the implementation of prevention and control measures, when the termination condition of emergency response is reached, the emergency response shall be terminated, and the early warning shall be lifted.

0.0069
Suggestions on terminating emergency response and releasing early warnings shall be made to the health administrative department when the emergency plans of foreign units are involved, and the termination conditions of emergency response are met.
0.0068 (Continued ) with the largest weight. In this study, "field investigation and control" has the greatest weight. The weight coefficient variation range of this criterion was 0.196-0.589, i.e., from -50%~50%, with a 10% correction each time [38]; the value of this proportion change was allocated to other standards in proportion. This evaluation system was applied to score several public health emergency drills (A1-A8) with existing online public data and then determine the grade change of alternative schemes through the change in the "field investigation and control" index weight. The ranking of alternatives with different weight values is shown in Table 6. By analyzing the results under different scenarios, it can be seen that the ranking of schemes has changed. Table 6 shows that the scheme ranking changes greatly under scenarios 0~10%. Under the scenario of -50%~-10%, the change in scheme ranking is small. The theoretical analysis is statistically verified by using Spearman correlation coefficient grade analysis [39], where Di represents the difference of ranks in a given scenario, and N is the number of pairs of ranks. The values of the Spearman correlation coefficient are given in the Table 7, and the results of "field investigation and control" under different weights are compared.
It can be seen from the table that the Spearman coefficient is 0.867,1, and the correlation degree is very high. It shows that the developed model has little effect on the final ranking under the change of weight coefficient, so it has good applicability.

Discussion
This study constructs n index system and determines the weight of factors through the Delphi and AHP, focuses on assessing the epidemic situation research, judging the situation and determining the management capacity of disease control personnel, and it provides more technical links and fewer process links with which to assess emergency response drills.
In the first round of consultation, many experts put forward suggestions on the modification of indicators, including the specialization of terms and ways to more accurately determine the scope of conditions, which resulted in deleted and added indicators. According to China's national conditions and the division of responsibilities among health department personnel, the function of epidemic situation release is in the health administrative department, not in disease control. Therefore, the "timely and active release of information to the public" by disease control officials was deleted. The CDC can only initiate plans for its own unit, so this part of the plan was modified.
Research shows that emergency preparedness, especially material preparation, is the key link between public health emergencies and emergency response. In the first round of consultation, more than two experts believed that the material preparation should be further subdivided, and the weight should be increased. Therefore, combined with practical applications, the materials are further divided into communication equipment, life support materials and professional materials.
In addition, in combination with the focus of work at this stage, the "daily meeting and daily log" in the handling of epidemic situations are necessary in such situations, so experts suggested adding biological samples to play a decisive role in the diagnosis of cases, as well as the discovery of atypical cases and asymptomatic infections. Therefore, experts also suggested increasing the requirements for biological transportation of infectious disease samples.
At present, it is easy to ignore the description of the AHP in most of the Delphi analytic hierarchy research [16]. As the AHP is an important step in determining the index weight and

Advantages and limitations of this study
Compared with the current no-notice drill mode evaluation system for public health emergencies, the research subjects are hospitals, and the research subjects of the evaluation system developed in this study are public health departments such as the CDC, which makes up for the gap in this regard. At present, the no-notice drill mode evaluation system for public health emergencies is basically constructed by the literature research method. This study adopts the Delphi AHP, which is more scientific. However, compared with most AHP studies in recent years, this study has more secondary indicators, resulting in too many pairwise comparisons in the judgment matrix, which may have some information bias.

Conclusion
The purpose of this study is to construct a set of evaluation index system of no-notice drills for public health emergencies is very important to improving the ability of public health department personnel to study, judge and deal with disease outbreaks like the pandemic. This is the first evaluation system to specifically query epidemiological investigators in the drill and research of a no-notice drill for public health emergencies, which is of great practical value to the CDC and other public health departments. The index focuses on the assessment of epidemiological investigation thinking, which plays an important role in improving the investigation and elimination ability of unexplained infectious diseases to give full play to the effect of emergency drills. In this study, the Delphi method is used to screen the indexes, and the weight of each index is determined by the AHP and percentage method. It is more scientific than the literature analysis method used in most of the current research on drill evaluation systems. Through Spearman rank correlation sensitivity analysis, it is found that under the change in weight coefficient, the change in scheme ranking is small, which further shows the stability of the results.
As the scope of public health emergencies is still large, in future research, we will modify and conduct in-depth research on the more important epidemic situation of infectious diseases (such as  in terms of the no-notice drill content of public health emergencies without to further improve the practicability and operability. When determining factor weights, several methods popular in the latest research have included the AHP, fuzzy evaluation method and the best-worst method; the calculation comparison is carried out at the same time to improve the scientificity and stability of the results.
Supporting information S1 File.